Papers with miscalibration problem
On Uncertainty Calibration and Selective Generation in Probabilistic Neural Summarization: A Benchmark Study (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Modern deep models for summarization generate miscalibrated predictive uncertainty, compromising reliability and trustworthiness in real-world applications. |
| Approach: | They propose to use probabilistic methods to improve the uncertainty quality of neural summarization models by using three large-scale benchmarks with varying difficulty. |
| Outcome: | The proposed methods consistently improve the model’s generation and uncertainty quality, leading to improved selective generation performance (i.e., abstaining from low-quality summaries) in practice. |
Large Language Models are Miscalibrated In-Context Learners (2025.findings-acl)
Copied to clipboard
| Challenge: | In-context Learning and Supervised Fine-Tuning have emerged as pre-dominant methodologies for machine learning and NLP. |
| Approach: | They propose to use self-ensembling to improve both performance and calibration of language models. |
| Outcome: | The proposed learning paradigms can achieve better calibration and better performance than the previous learning paradigm. |